Adaptive coding unit (cu) partitioning based on image statistics

ABSTRACT

A method for determining coding unit (CU) partitioning of a largest coding unit (LCU) of a picture is provided that includes computing a first statistical measure and a second statistical measure for the LCU, selecting the LCU as the CU partitioning when the first statistical measure does not exceed a first threshold and the second statistical measure does not exceed a second threshold, and selecting CUs in one or more lower layers of a CU hierarchy of the LCU to form the CU partitioning when the first statistical measure exceeds the first threshold and/or the second statistical measure exceeds the second threshold.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is a continuation of U.S. application Ser. No.14/083,423 filed Nov. 18, 2013, which claims benefit of U.S. ProvisionalPatent Application Ser. No. 61/727,938 filed Nov. 19, 2012, which areboth incorporated herein by reference in its entirety.

BACKGROUND OF THE INVENTION Field of the Invention

Embodiments of the present invention generally relate video coding andmore specifically relate to adaptive coding unit (CU) partitioning basedon image statistics.

Description of the Related Art

The Joint Collaborative Team on Video Coding (JCT-VC) of ITU-T WP3/16and ISO/IEC JTC 1/SC 29/WG 11 is currently developing thenext-generation video coding standard referred to as High EfficiencyVideo Coding (HEVC). Similar to previous video coding standards such asH.264/AVC, HEVC is based on a hybrid coding scheme using block-basedprediction and transform coding. First, the input signal is split intorectangular blocks that are predicted from the previously decoded databy either motion compensated (inter) prediction or intra prediction. Theresulting prediction error is coded by applying block transforms basedon an integer approximation of the discrete cosine transform, which isfollowed by quantization and entropy coding of the transformcoefficients.

HEVC is expected to provide around 50% improvement in coding efficiencyover the current standard, H.264/AVC, as well as larger resolutions andhigher frame rates. To address these requirements, HEVC utilizes largerblock sizes than the current video coding standard, H.264/AVC. Morespecifically, in HEVC, a largest coding unit (LCU) is the base unit usedfor block-based coding. An LCU plays a similar role in coding as the16×16 macroblock of H.264/AVC, but it may be larger, e.g., 32×32 or64×64. In HEVC, a picture is divided into non-overlapping LCUs. Tomaximize coding efficiency, each LCU may be partitioned into codingunits (CU) of different sizes using recursive quadtree partitioning. Themaximum hierarchical depth of the quadtree is determined by the size ofthe smallest CU (SCU) permitted. The quadtree partitioning of an LCUinto CUs is determined by a video encoder during prediction based on,e.g., minimization of rate/distortion costs.

To achieve the best encoding performance, an HEVC encoder should performan exhaustive search that considers all allowable CU sizes and allprediction modes for each CU size to select the best CU hierarchy for anLCU and the best prediction mode for each CU in the LCU, e.g., the CUhierarchy and prediction modes that produce the minimal rate-distortion(coding) cost. This exhaustive approach adds significant computationalcomplexity to the prediction process in an encoder. Real-time encodersmay have limited computational resources that do not allow for such anexhaustive search.

SUMMARY

Embodiments of the present invention relate to methods, apparatus, andcomputer readable media for adaptive CU partitioning based on imagestatistics is provided. In one aspect, a method for determining codingunit (CU) partitioning of a largest coding unit (LCU) of a picture isprovided that includes computing a first statistical measure and asecond statistical measure for the LCU, selecting the LCU as the CUpartitioning when the first statistical measure does not exceed a firstthreshold and the second statistical measure does not exceed a secondthreshold, and selecting CUs in one or more lower layers of a CUhierarchy of the LCU to form the CU partitioning when the firststatistical measure exceeds the first threshold and/or the secondstatistical measure exceeds the second threshold.

In one aspect, an apparatus configured to determine coding unit (CU)partitioning of a largest coding unit (LCU) of a picture is providedthat includes means for computing a first statistical measure and asecond statistical measure for the LCU, means for selecting the LCU asthe CU partitioning when the first statistical measure does not exceed afirst threshold and the second statistical measure does not exceed asecond threshold, and means for selecting CUs in one or more lowerlayers of a CU hierarchy of the LCU to form the CU partitioning when thefirst statistical measure exceeds the first threshold and/or the secondstatistical measure exceeds the second threshold.

In one aspect, a computer readable medium storing software instructionsis provided. The software instructions, when executed by at least oneprocessor, cause a method for determining coding unit (CU) partitioningof a largest coding unit (LCU) of a picture to be performed. The methodincludes computing a first statistical measure and a second statisticalmeasure for the LCU, selecting the LCU as the CU partitioning when thefirst statistical measure does not exceed a first threshold and thesecond statistical measure does not exceed a second threshold, andselecting CUs in one or more lower layers of a CU hierarchy of the LCUto form the CU partitioning when the first statistical measure exceedsthe first threshold and/or the second statistical measure exceeds thesecond threshold.

BRIEF DESCRIPTION OF THE DRAWINGS

Particular embodiments will now be described, by way of example only,and with reference to the accompanying drawings:

FIG. 1 is an example of quadtree based largest coding unit (LCU)decomposition;

FIG. 2 is an example;

FIG. 3 is a block diagram of a digital system;

FIGS. 4A and 4B are block diagrams of a video encoder;

FIG. 5 is a flow diagram of a method for adaptive coding unit (CU)partitioning;

FIG. 6 is an example; and

FIG. 7 is a block diagram of an illustrative digital system.

DETAILED DESCRIPTION OF EMBODIMENTS OF THE INVENTION

Specific embodiments of the invention will now be described in detailwith reference to the accompanying figures. Like elements in the variousfigures are denoted by like reference numerals for consistency.

As used herein, the term “picture” may refer to a frame or a field of aframe. A frame is a complete image captured during a known timeinterval. For convenience of description, embodiments of the inventionare described herein in reference to HEVC. One of ordinary skill in theart will understand that embodiments of the invention are not limited toHEVC.

Various versions of HEVC are described in the following documents, whichare incorporated by reference herein: T. Wiegand, et al., “WD3: WorkingDraft 3 of High-Efficiency Video Coding,” JCTVC-E603, JointCollaborative Team on Video Coding (JCT-VC) of ITU-T SG16 WP3 andISO/IEC JTC1/SC29/WG11, Geneva, CH, Mar. 16-23, 2011 (“WD3”), B. Bross,et al., “WD4: Working Draft 4 of High-Efficiency Video Coding,”JCTVC-F803_d6, Joint Collaborative Team on Video Coding (JCT-VC) ofITU-T SG16 WP3 and ISO/IEC JTC1/SC29/WG11, Torino, IT, Jul. 14-22, 2011(“WD4”), B. Bross. et al., “WD5: Working Draft 5 of High-EfficiencyVideo Coding,” JCTVC-G1103_d9, Joint Collaborative Team on Video Coding(JCT-VC) of ITU-T SG16 WP3 and ISO/IEC JTC1/SC29/WG11, Geneva, CH, Nov.21-30, 2011 (“WD5”), B. Bross, et al., “High Efficiency Video Coding(HEVC) Text Specification Draft 6,” JCTVC-H1003_dK, Joint CollaborativeTeam on Video Coding (JCT-VC) of ITU-T SG16 WP3 and ISO/IECJTC1/SC29/WG1, San Jose, Calif., Feb. 1-10, 2012, (“HEVC Draft 6”), B.Bross, et al., “High Efficiency Video Coding (HEVC) Text SpecificationDraft 7,” JCTVC-I1003_d9, Joint Collaborative Team on Video Coding(JCT-VC) of ITU-T SG16 WP3 and ISO/IEC JTC1/SC29/WG1, Geneva, CH, Apr.17-May 7, 2012 (“HEVC Draft 7”), B. Bross, et al., “High EfficiencyVideo Coding (HEVC) Text Specification Draft 8,” JCTVC-J1003_d7, JointCollaborative Team on Video Coding (JCT-VC) of ITU-T SG16 WP3 andISO/IEC JTC1/SC29/WG1, Stockholm, SE, Jul. 11-20, 2012 (“HEVC Draft 8”),B. Bross, et al., “High Efficiency Video Coding (HEVC) TextSpecification Draft 9,” JCTVC-K1003_v13, Joint Collaborative Team onVideo Coding (JCT-VC) of ITU-T SG16 WP3 and ISO/IEC JTC1/SC29/WG1,Shanghai, CN, Oct. 10-19, 2012 (“HEVC Draft 9”), and B. Bross, et al.,“High Efficiency Video Coding (HEVC) Text Specification Draft 10 (forFDIS & Last Call),” JCTVC-L1003_v34, Joint Collaborative Team on VideoCoding (JCT-VC) of ITU-T SG16 WP3 and ISO/IEC JTC1/SC29/WG1, Geneva, CH,Jan. 14-23, 2013 (“HEVC Draft 10”).

As previously mentioned, in HEVC, a largest coding unit (LCU) is thebase unit used for block-based coding. A picture is divided intonon-overlapping LCUs. That is, an LCU plays a similar role in coding asthe macroblock of H.264/AVC, but it may be larger, e.g., 32×32, 64×64,etc. An LCU may be partitioned into coding units (CU) using recursivequadtree partitioning. A CU is a block of pixels within an LCU and theCUs within an LCU may be of different sizes. The quadtree is splitaccording to various criteria until a leaf is reached, which is referredto as the coding node or coding unit. The maximum hierarchical depth ofthe quadtree is determined by the size of the smallest CU (SCU)permitted. The coding node is the root node of two trees, a predictiontree and a transform tree. A prediction tree specifies the position andsize of prediction units (PU) for a coding unit. A transform treespecifies the position and size of transform units (TU) for a codingunit. A transform unit may not be larger than a coding unit and the sizeof a transform unit may be, for example, 4×4, 8×8, 16×16, and 32×32. Thesizes of the transforms units and prediction units for a CU aredetermined by the video encoder during prediction based on minimizationof rate/distortion costs.

FIG. 1 shows an example of CU partitioning in which the LCU size is64×64 and the maximum hierarchical depth is 3. The recursive structure,i.e., the partitioning, is represented by a series of split flags. ForCU_(d), which has depth d and size 2N×2N, the coding of the CU isperformed in the current depth when split flag is set to zero. When thesplit flag is set to 1, CU_(d) is split into 4 independent CU_(d+1)which have depth (d+1) and size N×N. In this case, CU_(d+1) is referredto as a sub-CU of CU_(d). Unless the depth of sub-CU (d+1) is equal tothe maximum allowed depth, each CU_(d+1) is processed in a recursivemanner. If the depth of sub-CU (d+1) is equal to the maximum alloweddepth, further splitting is not allowed. For coding, a CU can be furthersplit into PUs and TUs. The sizes of an LCU and SCU are specified in theSequence Parameter Set (SPS). The embedded information in the SPS is LCUsize (s) and the maximum hierarchical depth (h) in a LCU. For example,if s=64 and h=4, then 4 CU sizes are possible: 64×64 (LCU), 32×32, 16×16and 8×8 (SCU). If s=16 and h=2, then 16×16 (LCU) and 8×8 (SCU) arepossible.

As previously mentioned, in some encoders, determination of the best CUstructure and the best prediction mode (intra or inter) for a CU isperformed bottom up, i.e., starting with the smallest possible CUpartitioning and working up the hierarchy levels (layers). Morespecifically, for each CU of each CU size, starting with the smallest CUsize, the encoder determines an intra-prediction coding cost for the CU,the best PU partition type for the CU, and an intra-prediction mode foreach PU of the best partition type. Similarly, for each CU of each CUsize, starting with the smallest CU size, the encoder determines aninter-prediction coding cost for the CU, the best PU partition type forthe CU, and an inter-prediction mode for each PU of the best partitiontype. Put another way, the best CU partition sizes are selected byexamining all CU layers in the hierarchy. First, the costs of CUs atlowest CU layer (smallest CU or SCU layer) are evaluated, and then costof each parent CU (of the SCUs) is evaluated, and compared to the costof SCUs. This process is repeated until largest CU (LCU) layer isreached. FIG. 2 is an example of this CU partitioning method in whichthe SCU size is assumed to be 16×16 and the LCU size is assumed to be64×64. This exhaustive approach for determining prediction modes, whileproviding the best encoding efficiency, adds significant computationalcomplexity to the prediction process in an encoder.

Embodiments of the invention provide top down adaptive CU partitioningof an image based on image statistics. Rather than determining thepartitioning of an image using the bottom up exhaustive search based oncoding costs of the prior art, the partitioning is determined top downand without consideration of coding costs and prediction modes. Once thepartitioning is determined, prediction modes are then determined for theselected partitions. In general, for a given CU, two statisticalmeasures are computed. If both measures do not exceed correspondingthresholds, the CU is selected as a partition of the image. If eithermeasure exceeds the corresponding threshold, the CU is partitioned intochild CUs, and the process is repeated for each child CU. In someembodiments, the two statistical measures are the variance of the CU andthe gradient of the CU.

FIG. 3 shows a block diagram of a digital system that includes a sourcedigital system 300 that transmits encoded video sequences to adestination digital system 302 via a communication channel 316. Thesource digital system 300 includes a video capture component 304, avideo encoder component 306, and a transmitter component 308. The videocapture component 304 is configured to provide a video sequence to beencoded by the video encoder component 306. The video capture component304 may be, for example, a video camera, a video archive, or a videofeed from a video content provider. In some embodiments, the videocapture component 304 may generate computer graphics as the videosequence, or a combination of live video, archived video, and/orcomputer-generated video.

The video encoder component 306 receives a video sequence from the videocapture component 304 and encodes it for transmission by the transmittercomponent 308. The video encoder component 306 receives the videosequence from the video capture component 304 as a sequence of pictures,divides the pictures into largest coding units (LCUs), and encodes thevideo data in the LCUs. As part of the encoding process, the videoencoder component 306 may perform adaptive CU partitioning as describedherein. An embodiment of the video encoder component 306 is described inmore detail herein in reference to FIGS. 4A and 4B.

The transmitter component 308 transmits the encoded video data to thedestination digital system 302 via the communication channel 316. Thecommunication channel 316 may be any communication medium, orcombination of communication media suitable for transmission of theencoded video sequence, such as, for example, wired or wirelesscommunication media, a local area network, or a wide area network.

The destination digital system 302 includes a receiver component 310, avideo decoder component 312 and a display component 314. The receivercomponent 310 receives the encoded video data from the source digitalsystem 300 via the communication channel 316 and provides the encodedvideo data to the video decoder component 312 for decoding. The videodecoder component 312 reverses the encoding process performed by thevideo encoder component 306 to reconstruct the LCUs of the videosequence.

The reconstructed video sequence is displayed on the display component314. The display component 314 may be any suitable display device suchas, for example, a plasma display, a liquid crystal display (LCD), alight emitting diode (LED) display, etc.

In some embodiments, the source digital system 300 may also include areceiver component and a video decoder component and/or the destinationdigital system 302 may include a transmitter component and a videoencoder component for transmission of video sequences both directionsfor video steaming, video broadcasting, and video telephony. Further,the video encoder component 306 and the video decoder component 312 mayperform encoding and decoding in accordance with one or more videocompression standards. The video encoder component 306 and the videodecoder component 312 may be implemented in any suitable combination ofsoftware, firmware, and hardware, such as, for example, one or moredigital signal processors (DSPs), microprocessors, discrete logic,application specific integrated circuits (ASICs), field-programmablegate arrays (FPGAs), etc.

FIGS. 4A and 4B show block diagrams of an example video encoderconfigured to perform adaptive CU partitioning based on CU statistics aspart of the encoding process. FIG. 4A shows a high level block diagramof the video encoder and FIG. 4B shows a block diagram of the LCUprocessing component 442 of the video encoder. As shown in FIG. 4A, thevideo encoder includes a coding control component 440, an LCUpartitioning component 441, an LCU processing component 442, and amemory 446. The memory 446 may be internal (on-chip) memory, external(off-chip) memory, or a combination thereof. The memory 446 may be usedto communicate information between the various components of the videoencoder.

An input digital video sequence is provided to the coding controlcomponent 440. The coding control component 440 sequences the variousoperations of the video encoder, i.e., the coding control component 440runs the main control loop for video encoding. For example, the codingcontrol component 440 performs processing on the input video sequencethat is to be done at the picture level, such as determining the codingtype (I, P, or B) of a picture based on a high level coding structure,e.g., IPPP, IBBP, hierarchical-B, and dividing a picture into LCUs forfurther processing.

In addition, for pipelined architectures in which multiple LCUs may beprocessed concurrently in different components of the LCU processing,the coding control component 440 controls the processing of the LCUs byvarious components of the LCU processing in a pipeline fashion. Forexample, in many embedded systems supporting video processing, there maybe one master processor and one or more slave processing modules, e.g.,hardware accelerators. The master processor operates as the codingcontrol component and runs the main control loop for video encoding, andthe slave processing modules are employed to off load certaincompute-intensive tasks of video encoding such as motion estimation,motion compensation, intra prediction mode estimation, transformationand quantization, entropy coding, and loop filtering. The slaveprocessing modules are controlled in a pipeline fashion by the masterprocessor such that the slave processing modules operate on differentLCUs of a picture at any given time. That is, the slave processingmodules are executed in parallel, each processing its respective LCUwhile data movement from one processor to another is serial.

The LCU partitioning component 441 determines the CU partitioning ofeach LCU and provides this partitioning to the LCU processing component442. More specifically, the LCU partitioning component 441 performs anembodiment of the CU partitioning method of FIG. 5 on each LCU todetermine the partitioning for encoding.

FIG. 4B is a block diagram of the LCU processing component 442. The LCUprocessing component 442 receives LCUs 400 of the input video sequencefrom the coding control component 440 and the CU partitioning of each ofthe LCUs 400 from the LCU partitioning component 441 and encodes theLCUs 400 under the control of the coding control component 440 togenerate the compressed video stream. The LCUs 400 from the codingcontrol component 440 and the CU partitioning for each LCU are providedas inputs to the motion estimation component (ME) 420 and to theintra-prediction estimation component (IPE) 424. The LCUs are alsoprovided to a positive input of a combiner 402 (e.g., adder orsubtractor or the like). Further, although not specifically shown, theprediction mode of each picture as selected by the coding controlcomponent 440 is provided to a mode decision component 428 and theentropy coding component 436.

The storage component 418 provides reference data to the motionestimation component 420 and to the motion compensation component 422.The reference data may include one or more previously encoded anddecoded pictures, i.e., reference pictures.

The motion estimation component 420 provides motion data information tothe motion compensation component 422 and the entropy coding component436. More specifically, the motion estimation component 420 performstests on CUs in an LCU (as per the CU partitioning determined by the LCUpartitioning component 441) based on multiple inter-prediction modes(e.g., skip mode, merge mode, and normal or direct inter-prediction), PUsizes, and TU sizes using reference picture data from storage 418 tochoose the best PU/TU partitioning, inter-prediction modes, motionvectors, etc. for each CU based on coding cost, e.g., a rate distortioncoding cost. To perform the tests, the motion estimation component 420may divide each CU into PUs according to the unit sizes of theinter-prediction modes and into TUs according to the transform unitsizes, and calculate the coding costs for each PU size, prediction mode,and transform unit size for each CU. The motion estimation component 420provides the motion vector (MV) or vectors and the prediction mode foreach PU in the CU partitioning to the motion compensation component (MC)422.

The motion compensation component 422 receives the selectedinter-prediction mode and mode-related information from the motionestimation component 420 and generates the inter-predicted CUs. Theinter-predicted CUs are provided to the mode decision component 428along with the selected inter-prediction modes for the inter-predictedPUs and corresponding TU sizes for the selected PU/TU partitioning. Thecoding costs of the inter-predicted CUs are also provided to the modedecision component 428.

The intra-prediction estimation component 424 (IPE) performsintra-prediction estimation in which tests on CUs in an LCU (as per theCU partitioning determined by the LCU partitioning component 441) basedon multiple intra-prediction modes, PU sizes, and TU sizes are performedusing reconstructed data from previously encoded neighboring CUs storedin a buffer (not shown) to choose the best PU/TU partitioning andintra-prediction modes based on coding cost, e.g., a rate distortioncoding cost. To perform the tests, the intra-prediction estimationcomponent 424 may divide each CU into PUs according to the unit sizes ofthe intra-prediction modes and into TUs according to the transform unitsizes, and calculate the coding costs for each PU size, prediction mode,and transform unit size for each PU. The intra-prediction estimationcomponent 424 provides the selected intra-prediction modes for the PUs,and the corresponding TU sizes for each CU in the CU partitioning to theintra-prediction component (IP) 426. The coding costs of theintra-predicted CUs are also provided to the intra-prediction component426.

The intra-prediction component 426 (IP) receives intra-predictioninformation, e.g., the selected mode or modes for the PU(s), the PUsize, etc., from the intra-prediction estimation component 424 andgenerates the intra-predicted CUs. The intra-predicted CUs are providedto the mode decision component 428 along with the selectedintra-prediction modes for the intra-predicted PUs and corresponding TUsizes for the selected PU/TU partitioning. The coding costs of theintra-predicted CUs are also provided to the mode decision component428.

The mode decision component 428 selects between intra-prediction of a CUand inter-prediction of a CU based on the intra-prediction coding costof the CU from the intra-prediction component 426, the inter-predictioncoding cost of the CU from the motion compensation component 422, andthe picture prediction mode provided by the coding control component440. Based on the decision as to whether a CU is to be intra- orinter-coded, the intra-predicted PUs or inter-predicted PUs areselected. The selected CU/PU/TU partitioning with corresponding modesand other mode related prediction data (if any) such as motion vector(s)and reference picture index (indices), are provided to the entropycoding component 436.

The output of the mode decision component 428, i.e., the predicted PUs,is provided to a negative input of the combiner 402 and to the combiner438. The associated transform unit size is also provided to thetransform component 404. The combiner 402 subtracts a predicted PU fromthe original PU. Each resulting residual PU is a set of pixel differencevalues that quantify differences between pixel values of the original PUand the predicted PU. The residual blocks of all the PUs of a CU form aresidual CU for further processing.

The transform component 404 performs block transforms on the residualCUs to convert the residual pixel values to transform coefficients andprovides the transform coefficients to a quantize component 406. Morespecifically, the transform component 404 receives the transform unitsizes for the residual CU and applies transforms of the specified sizesto the CU to generate transform coefficients. Further, the quantizecomponent 406 quantizes the transform coefficients based on quantizationparameters (QPs) and quantization matrices provided by the codingcontrol component 440 and the transform sizes and provides the quantizedtransform coefficients to the entropy coding component 436 for coding inthe bit stream.

The entropy coding component 436 entropy encodes the relevant data,i.e., syntax elements, output by the various encoding components and thecoding control component 440 to generate the compressed video bitstream. Among the syntax elements that are encoded are picture parametersets, flags indicating the CU/PU/TU partitioning of an LCU, theprediction modes for the CUs, and the quantized transform coefficientsfor the CUs.

The LCU processing component 442 includes an embedded decoder. As anycompliant decoder is expected to reconstruct an image from a compressedbit stream, the embedded decoder provides the same utility to the videoencoder. Knowledge of the reconstructed input allows the video encoderto transmit the appropriate residual energy to compose subsequentpictures.

The quantized transform coefficients for each CU are provided to aninverse quantize component (IQ) 412, which outputs a reconstructedversion of the transform result from the transform component 404. Thedequantized transform coefficients are provided to the inverse transformcomponent (IDCT) 414, which outputs estimated residual informationrepresenting a reconstructed version of a residual CU. The inversetransform component 414 receives the transform unit size used togenerate the transform coefficients and applies inverse transform(s) ofthe specified size to the transform coefficients to reconstruct theresidual values. The reconstructed residual CU is provided to thecombiner 438.

The combiner 438 adds the original predicted CU to the residual CU togenerate a reconstructed CU, which becomes part of reconstructed picturedata. The reconstructed picture data is stored in a buffer (not shown)for use by the intra-prediction estimation component 424.

Various in-loop filters may be applied to the reconstructed picture datato improve the quality of the reference picture data used forencoding/decoding of subsequent pictures. The in-loop filters mayinclude a deblocking filter 430, a sample adaptive offset filter (SAO)432, and an adaptive loop filter (ALF) 434. The in-loop filters 430,432, 434 are applied to each reconstructed LCU in the picture and thefinal filtered reference picture data is provided to the storagecomponent 418. In some embodiments, the ALF component 434 is notpresent.

FIG. 5 is a flow diagram of a method for adaptive CU partitioning of anLCU based on image statistics. The method may be executed by an encodersuch as that of FIGS. 4A and 4B to determine the CU partitioning ofLCUs. For simplicity of explanation, embodiments are described hereinassuming a 64×64 LCU and a hierarchy depth of 2. Thus, the smallest CUsize is 16×16 and there are three layers in the hierarchy. One ofordinary skill in the art will understand other embodiments withdiffering LCU sizes and/or hierarchy depths.

The method determines the CU partitioning of an LCU top down, i.e.,beginning at the top level (layer) of the CU hierarchy. For a given CU,the variance and gradient are computed and compared to variance andgradient thresholds, referred to respectively as ThrVar(n) andThrGrad(n), where n is the index of the CU layer in the hierarchy. Forexample, n=0 indicates that the layer is the LCU layer, n=1 indicatesthat the CU layer is the 32×32 CU layer, and n=2 indicates that the CUlayer is the 16×16 CU layer (SCU layer). If the variance and gradientare not larger than the respective thresholds, the CU is selected as apartition of the LCU. If the variance or gradient is larger than therespective threshold, the process is repeated for all child CUs of theCU, i.e., for child CUs at the (n+1) CU layer. Each threshold may haveany suitable value and that value may be determined experimentally basedon, for example, the particular application and/or expected content ofvideo streams to be encoded.

As shown in FIG. 5, initially the variance and gradient of the LCU iscomputed 500. In general, variance is a measure of the dispersion ofpixels values in a block of pixels around the mean pixel value of theblock. Variance may be computed as the sum of the squared differencesbetween each pixel in a block of pixels and the average pixel value ofthe block, i.e.,

var=Σ_(m=0) ^(m−1)(p _(m) −p _(avg))²

where m is the number of pixels in the block and p_(avg) is the averagepixel value of the block.

In general, gradient is a measure of the directional change in theintensity or color of a block of pixels. The gradient includes both ahorizontal gradient and a vertical gradient. The horizontal gradient ofa block of pixels may be computed as the sum of the differences betweeneach pixel in a block of pixels and the horizontally neighboring pixel,i.e.,

$h = {\sum\limits_{{col} = 0}^{{col} = {s - 2}}{\sum\limits_{{row} = 0}^{{row} = {s - 1}}\;{{p_{{row},{col}} - p_{{row},{{col} + 1}}}}}}$

where s is the block size, e.g., 8, 16, 32, 64. The vertical gradientmay be computed as the sum of the differences between each pixel in theblock of pixels and the vertically neighboring pixel, i.e.,

$v = {\sum\limits_{{row} = 0}^{{row} = {s - 2}}\;{\sum\limits_{{col} = 0}^{{col} = {s - 1}}\;{{p_{{row},{col}} - p_{{{row} + 1},{col}}}}}}$

where s is the block size, e.g., 8, 16, 32, 64. The gradient of theblock is the sum of the horizontal gradient h and the vertical gradientv.

The variance and gradient computed for the LCU are then compared 502 toa variance threshold (ThrVar(0)) and a gradient threshold (ThrGrad(0))for the LCU layer of the hierarchy. If both the variance and thegradient are less than or equal to the respective thresholds, the LCUpartition is selected as the partitioning of the LCU and processingterminates. If one or both of the variance and the gradient are greaterthan the respective thresholds, processing continues with the four 32×32child CUs of the LCU (504-512).

The variance and gradient of a 32×32 child CU is computed 504. Thevariance and gradient computed for the 32×32 child CU are then compared506 to a variance threshold (ThrVar(1)) and a gradient threshold(ThrGrad(1)) for the 32×32 CU layer of the hierarchy. If both thevariance and the gradient are less than or equal to the respectivethresholds, the 32×32 partition is selected 508 as the part of thepartitioning of the LCU and processing continue with the next 32×32child CU, if any 512. If one or both of the variance and the gradientare greater than the respective thresholds, the 16×16 child CUs of the32×32 CU are selected 510 as partitions of the LCU, and processingcontinues with the next 32×32 child CU, if any 512.

After the four 32×32 CUs are processed, the process terminated and theCU partitioning is returned for further processing in the encodingprocess. As was previously described, the encoder will then determinethe best prediction mode and PU/TU configuration for each CU in the CUpartitioning and encode the CUs according to the selected predictionmodes and PU/TU configurations.

FIG. 6 is an example illustrating the method of FIG. 5. In this example,first the variance and gradient are computed for the LCU. Either one orboth of these values is greater than the corresponding thresholds forthe LCU layer, so the LCU is not selected as the CU partitioning and thefour 32×32 child CU are selected as a possible partitioning. Each the32×32 child CUs is then processed to determine if the particular CUshould be a partition of the LCU. For each of the 32×32 child CUs, agradient and threshold is computed and are compared to the respectivecorresponding thresholds for the 32×32 CU layer. The variance andgradient of the top right and bottom right 32×32 child CUs are both lessthan the corresponding thresholds, so these 32×32 CUs are selected aspart of the CU partitioning of the LCU. One or both of the variance andgradient of the other two 32×32 child CUs exceeds the correspondingthreshold, so the 16×16 child CUs of these 32×32 CUs are selected aspart of the partitioning of the LCU.

FIG. 7 is a block diagram of an example digital system suitable for useas an embedded system that may be configured to encode a video sequenceusing a method for adaptive CU partitioning based on image statistics asdescribed herein. This example system-on-a-chip (SoC) is representativeof one of a family of DaVinci™ Digital Media Processors, available fromTexas Instruments, Inc. This SoC is described in more detail in“TMS320DM6467 Digital Media System-on-Chip”, SPRS403G, Dec. 2007 orlater, which is incorporated by reference herein.

The SoC 700 is a programmable platform designed to meet the processingneeds of applications such as video encode/decode/transcode/transrate,video surveillance, video conferencing, set-top box, medical imaging,media server, gaming, digital signage, etc. The SoC 700 provides supportfor multiple operating systems, multiple user interfaces, and highprocessing performance through the flexibility of a fully integratedmixed processor solution. The device combines multiple processing coreswith shared memory for programmable video and audio processing with ahighly-integrated peripheral set on common integrated substrate.

The dual-core architecture of the SoC 700 provides benefits of both DSPand Reduced Instruction Set Computer (RISC) technologies, incorporatinga DSP core and an ARM926EJ-S core. The ARM926EJ-S is a 32-bit RISCprocessor core that performs 32-bit or 16-bit instructions and processes32-bit, 16-bit, or 8-bit data. The DSP core is a TMS320C64x+TM core witha very-long-instruction-word (VLIW) architecture. In general, the ARM isresponsible for configuration and control of the SoC 700, including theDSP Subsystem, the video data conversion engine (VDCE), and a majorityof the peripherals and external memories. The switched central resource(SCR) is an interconnect system that provides low-latency connectivitybetween master peripherals and slave peripherals. The SCR is thedecoding, routing, and arbitration logic that enables the connectionbetween multiple masters and slaves that are connected to it.

The SoC 700 also includes application-specific hardware logic, on-chipmemory, and additional on-chip peripherals. The peripheral set includes:a configurable video port (Video Port I/F), an Ethernet MAC (EMAC) witha Management Data Input/Output (MDIO) module, a 4-bit transfer/4-bitreceive VLYNQ interface, an inter-integrated circuit (I2C) businterface, multichannel audio serial ports (McASP), general-purposetimers, a watchdog timer, a configurable host port interface (HPI);general-purpose input/output (GPIO) with programmable interrupt/eventgeneration modes, multiplexed with other peripherals, UART interfaceswith modem interface signals, pulse width modulators (PWM), an ATAinterface, a peripheral component interface (PCI), and external memoryinterfaces (EMIFA, DDR2). The video port I/F is a receiver andtransmitter of video data with two input channels and two outputchannels that may be configured for standard definition television(SDTV) video data, high definition television (HDTV) video data, and rawvideo data capture.

As shown in FIG. 7, the SoC 700 includes two high-definitionvideo/imaging coprocessors (HDVICP) and a video data conversion engine(VDCE) to offload many video and image processing tasks from the DSPcore. The VDCE supports video frame resizing, anti-aliasing, chrominancesignal format conversion, edge padding, color blending, etc. The HDVICPcoprocessors are designed to perform computational operations requiredfor video encoding and/or decoding such as motion estimation, motioncompensation, intra-prediction, transformation, inverse transformation,quantization, and inverse quantization. Further, the distinct circuitryin the HDVICP coprocessors that may be used for specific computationoperations is designed to operate in a pipeline fashion under thecontrol of the ARM subsystem and/or the DSP subsystem.

As was previously mentioned, the SoC 700 may be configured to encode avideo sequence using a method for adaptive CU partitioning based onimage statistics as described herein. For example, the coding controland the LCU partitioning of the video encoder of FIGS. 4A and 4B may beexecuted on the DSP subsystem or the ARM subsystem and at least some ofthe computational operations of the block processing, including theintra-prediction and inter-prediction of mode selection, transformation,quantization, and entropy encoding may be executed on the HDVICPcoprocessors.

Other Embodiments

While the invention has been described with respect to a limited numberof embodiments, those skilled in the art, having benefit of thisdisclosure, will appreciate that other embodiments can be devised whichdo not depart from the scope of the invention as disclosed herein.

For example, embodiments have been described herein in which each parentblock has four child blocks. One of ordinary skill in the art willunderstand embodiments in which the number of child blocks may differ.

In another example, embodiments have been described herein in whichvariance is used as one of the image statistics for determination of CUpartitioning. One of ordinary skill in the art will understandembodiments in which an alternative statistical measurement is used. Forexample, the previously described variance computation includesmultiplication, which will require more logic gates in a hardwareimplementation. Instead of computing variance, the simple sum ofabsolute differences between pixel values and the average pixel valuemay be computed. This latter statistic computation can be implementedwith fewer logic gates than the full variance computation.

In another example, embodiments have been described herein in which eachlayer of the CU hierarchy has corresponding gradient and variancethreshold values. One of ordinary skill in the art will understandembodiments in which threshold values are normalized and thus there mayonly be one threshold value for each statistical measure, e.g., onethreshold value for variance and one threshold value for gradient.Further, one of ordinary skill in the art will understand embodiments inwhich some layers may have unique threshold values while other layersshare threshold values.

In another example, embodiments have been described herein in whichgradient is used as one of the statistical measures in deciding the CUpartitioning, where gradient is defined as the sum of the horizontalgradient and the vertical gradient. One of ordinary skill in the artwill understand embodiments in which the horizontal and verticalgradients are used as statistical measures in lieu of adding them togenerate a single gradient. In such embodiments, suitable thresholds forthe horizontal and vertical gradients would be used.

Embodiments of the methods and encoders described herein may beimplemented in hardware, software, firmware, or any combination thereof.If completely or partially implemented in software, the software may beexecuted in one or more processors, such as a microprocessor,application specific integrated circuit (ASIC), field programmable gatearray (FPGA), or digital signal processor (DSP). The softwareinstructions may be initially stored in a computer-readable medium andloaded and executed in the processor. In some cases, the softwareinstructions may also be sold in a computer program product, whichincludes the computer-readable medium and packaging materials for thecomputer-readable medium. In some cases, the software instructions maybe distributed via removable computer readable media, via a transmissionpath from computer readable media on another digital system, etc.Examples of computer-readable media include non-writable storage mediasuch as read-only memory devices, writable storage media such as disks,flash memory, memory, or a combination thereof.

Although method steps may be presented and described herein in asequential fashion, one or more of the steps shown in the figures anddescribed herein may be performed concurrently, may be combined, and/ormay be performed in a different order than the order shown in thefigures and/or described herein. Accordingly, embodiments should not beconsidered limited to the specific ordering of steps shown in thefigures and/or described herein.

It is therefore contemplated that the appended claims will cover anysuch modifications of the embodiments as fall within the true scope ofthe invention.

What is claimed is:
 1. An encoder comprising: a coding componentconfigured to output a largest coding unit (LCU) of an image; and apartitioning component configured to: receive the LCU; determine a firststatistical measure of the LCU; determine a second statistical measureof the LCU; compare the first statistical measure to a first threshold;compare the second statistical measure to a second threshold; andresponsive to the first statistical measure exceeding the firstthreshold or the second statistical measure exceeding the secondthreshold, partition the LCU into a set of child coding units (CUs). 2.The encoder of claim 1, wherein: the first statistical measure is avariance; and the second statistical measure is a gradient.
 3. Theencoder of claim 1, wherein: the coding component is further configuredto: receive a digital video sequence; process the digital video sequenceinto a plurality of images; and divide each of the plurality of imagesinto respective LCUs; the LCU is one of the respective LCUs; and theimage is one of the plurality of images.
 4. The encoder of claim 3,wherein: the first threshold and the second threshold are based on anexpected content of the digital video sequence.
 5. The encoder of claim1, wherein: the coding component is further configured to: responsive tothe first statistical measure not exceeding the first threshold and thesecond statistical measure not exceeding the second threshold, selectthe LCU as a partition of the image.
 6. The encoder of claim 1, wherein:the coding component is further configured to: determine a thirdstatistical measure for a first child CU of the set of child CUs;determine a fourth statistical measure for the first child CU the set ofchild CUs; compare the third statistical measure to a third threshold;compare the fourth statistical measure to a fourth threshold; andresponsive to the third statistical measure exceeding the thirdthreshold or the fourth statistical measure exceeding the fourththreshold, partition the first child CU into a second set of child CUs.7. The encoder of claim 6, wherein: the coding component is furtherconfigured to: determine a fifth statistical measure for a second childCU of the set of child CUs; determine a sixth statistical measure forthe second child CU the set of child CUs; compare the fifth statisticalmeasure to the third threshold; compare the sixth statistical measure tothe fourth threshold; and responsive to the fifth statistical measureexceeding the third threshold or the sixth statistical measure exceedingthe fourth threshold, partition the second child CU into a third set ofchild CUs.
 8. A method comprising: receiving, by a processor, a largestcoding unit (LCU); determining, by the processor, a first statisticalmeasure of the LCU; determining, by the processor, a second statisticalmeasure of the LCU; comparing, by the processor, the first statisticalmeasure to a first threshold; comparing, by the processor, the secondstatistical measure to a second threshold; and responsive to the firststatistical measure exceeding the first threshold or the secondstatistical measure exceeding the second threshold, partitioning, by theprocessor, the LCU into a set of child coding units (CUs).
 9. The methodof claim 8, wherein: the first statistical measure is a variance; andthe second statistical measure is a gradient.
 10. The method of claim 8,further comprising: receiving, by the processor, a digital videosequence; processing, by the processor, the digital video sequence intoa plurality of images; and dividing, by the processor, each of theplurality of images into respective LCUs, wherein the LCU is one of therespective LCUs.
 11. The method of claim 10, wherein: the firstthreshold and the second threshold are based on an expected content ofthe digital video sequence.
 12. The method of claim 8, furthercomprising: responsive to the first statistical measure not exceedingthe first threshold and the second statistical measure not exceeding thesecond threshold, selecting, by the processor, the LCU as a partition ofan image.
 13. The method of claim 8, further comprising: determining, bythe processor, a third statistical measure for a first child CU of theset of child CUs; determining, by the processor, a fourth statisticalmeasure for the first child CU the set of child CUs; comparing, by theprocessor, the third statistical measure to a third threshold;comparing, by the processor, the fourth statistical measure to a fourththreshold; and responsive to the third statistical measure exceeding thethird threshold or the fourth statistical measure exceeding the fourththreshold, partitioning, by the processor, the first child CU into asecond set of child CUs.
 14. The method of claim 13, further comprising:determining, by the processor, a fifth statistical measure for a secondchild CU of the set of child CUs; determining, by the processor, a sixthstatistical measure for the second child CU the set of child CUs;comparing, by the processor, the fifth statistical measure to the thirdthreshold; comparing, by the processor, the sixth statistical measure tothe fourth threshold; and responsive to the fifth statistical measureexceeding the third threshold or the sixth statistical measure exceedingthe fourth threshold, partitioning, by the processor, the second childCU into a third set of child CUs.
 15. A computer readable medium storingsoftware instructions that, when executed by at least one processor,causes a method for determining coding unit (CU) partitioning of alargest coding unit (LCU) of a picture to be performed, the methodcomprising: determining a first statistical measure of the LCU;determining a second statistical measure of the LCU; comparing the firststatistical measure to a first threshold; comparing the secondstatistical measure to a second threshold; and responsive to the firststatistical measure exceeding the first threshold or the secondstatistical measure exceeding the second threshold, partitioning the LCUinto a set of child CUs.
 16. The computer readable medium of claim 15,wherein: the first statistical measure is a variance; and the secondstatistical measure is a gradient.
 17. The computer readable medium ofclaim 15, wherein the method further comprises: receiving a digitalvideo sequence; processing the digital video sequence into a pluralityof images; and dividing each of the plurality of images into respectiveLCUs, wherein the LCU is one of the respective LCUs.
 18. The computerreadable medium of claim 15, wherein the method further comprises:responsive to the first statistical measure not exceeding the firstthreshold and the second statistical measure not exceeding the secondthreshold, selecting, by the processor, the LCU as a partition of animage.
 19. The computer readable medium of claim 15, wherein the methodfurther comprises: determining, by the processor, a third statisticalmeasure for a first child CU of the set of child CUs; determining, bythe processor, a fourth statistical measure for the first child CU theset of child CUs; comparing, by the processor, the third statisticalmeasure to a third threshold; comparing, by the processor, the fourthstatistical measure to a fourth threshold; and responsive to the thirdstatistical measure exceeding the third threshold or the fourthstatistical measure exceeding the fourth threshold, partitioning, by theprocessor, the first child CU into a second set of child CUs.
 20. Thecomputer readable medium of claim 19, wherein the method furthercomprises: determining, by the processor, a fifth statistical measurefor a second child CU of the set of child CUs; determining, by theprocessor, a sixth statistical measure for the second child CU the setof child CUs; comparing, by the processor, the fifth statistical measureto the third threshold; comparing, by the processor, the sixthstatistical measure to the fourth threshold; and responsive to the fifthstatistical measure exceeding the third threshold or the sixthstatistical measure exceeding the fourth threshold, partitioning, by theprocessor, the second child CU into a third set of child CUs.